{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文本分块：当加载完一个文件之后，需要对文件进行分割，切割逻辑可以按照：章节、段落、句子、词组等\n",
    "\n",
    "1. 上下文长度的限制，LLM通常有固定的输入长度限制（4K、8K 、 16K、32K等）  GPT4 turbo 128K\n",
    "2. 提高处理效率，减少响应时间，\n",
    "3. 提高输出质量，LLM处理较短的文本块时，通常能够生成更精确、更相关的输出。这是因为模型更容易理解和响应较小的文本范围内的复杂性和细节。\n",
    "\n",
    "`TextSplitter`的工作 流程\n",
    "1. 将文本拆分成小的、语义上有意义的块（通常是句子）。\n",
    "2. 将这些小块组合成一个较大的块，直到达到一定的大小**chunksize**（通过某种函数来衡量,一般是长度）。\n",
    "3. 一旦达到这个大小，就将这个大块视为一个独立的文本片段，然后开始创建一个新的文本块，并保留一些重叠部分（**chunkoverlap**以保持块与块之间的上下文联系）。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "孔乙己\n",
    "\n",
    "孔乙己是站着喝酒而穿长衫的唯一的人。他身材很高大;青白脸色，绉纹间时常夹些伤痕;一部乱蓬蓬的花白的胡子。穿的虽然是长衫，可是又脏又破，似乎十多年没有补，也没有洗。\n",
    "他对人说话，总是满口之乎者也，教人半懂不懂的。因为他姓孔，别人便从描红纸上的“上大人孔乙己”这半懂不懂的话里，替他取下一个绰号，叫作孔乙己。\n",
    "\n",
    "孔乙己一到店，所有喝酒的人便都看着他笑，有的叫道，“孔乙己，你脸上又添上新伤疤了!”他不回答，对柜里说，“温两碗酒，要一碟茴香豆。”便排出九文大钱。他们又故意的高声嚷道，“你一定又偷了人家的东西了!”孔乙己睁大眼睛说，“你怎么这样凭空污人清白……”“什么清白?我前天亲眼见你偷了何家的书，吊着打。”\n",
    "孔乙己便涨红了脸，额上的青筋条条绽出，争辩道，“窃书不能算偷……窃书!……读书人的事，能算偷么?”接连便是难懂的话，什么“君子固穷”，什么“者乎”之类，引得众人都哄笑起来：店内外充满了快活的空气。听人家背地里谈论，孔乙己原来也读过书，但终于没有进学，又不会营生;于是愈过愈穷，弄到将要讨饭了。\n",
    "幸而写得一笔好字，便替人家钞钞书，换一碗饭吃。可惜他又有一样坏脾气，便是好喝懒做。坐不到几天，便连人和书籍纸张笔砚，一齐失踪。如是几次，叫他钞书的人也没有了。孔乙己没有法，便免不了偶然做些偷窃的事。但他在我们店里，品行却比别人都好，就是从不拖欠;虽然间或没有现钱，暂时记在粉板上，但不出一月，定然还清，从粉板上拭去了孔乙己的名字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = '''孔乙己\n",
    "\n",
    "孔乙己是站着喝酒而穿长衫的唯一的人。他身材很高大;青白脸色，绉纹间时常夹些伤痕;一部乱蓬蓬的花白的胡子。穿的虽然是长衫，可是又脏又破，似乎十多年没有补，也没有洗。\n",
    "他对人说话，总是满口之乎者也，教人半懂不懂的。因为他姓孔，别人便从描红纸上的“上大人孔乙己”这半懂不懂的话里，替他取下一个绰号，叫作孔乙己。\n",
    "\n",
    "孔乙己一到店，所有喝酒的人便都看着他笑，有的叫道，“孔乙己，你脸上又添上新伤疤了!”他不回答，对柜里说，“温两碗酒，要一碟茴香豆。”便排出九文大钱。他们又故意的高声嚷道，“你一定又偷了人家的东西了!”孔乙己睁大眼睛说，“你怎么这样凭空污人清白……”“什么清白?我前天亲眼见你偷了何家的书，吊着打。”\n",
    "孔乙己便涨红了脸，额上的青筋条条绽出，争辩道，“窃书不能算偷……窃书!……读书人的事，能算偷么?”接连便是难懂的话，什么“君子固穷”，什么“者乎”之类，引得众人都哄笑起来：店内外充满了快活的空气。听人家背地里谈论，孔乙己原来也读过书，但终于没有进学，又不会营生;于是愈过愈穷，弄到将要讨饭了。\n",
    "幸而写得一笔好字，便替人家钞钞书，换一碗饭吃。可惜他又有一样坏脾气，便是好喝懒做。坐不到几天，便连人和书籍纸张笔砚，一齐失踪。如是几次，叫他钞书的人也没有了。孔乙己没有法，便免不了偶然做些偷窃的事。但他在我们店里，品行却比别人都好，就是从不拖欠;虽然间或没有现钱，暂时记在粉板上，但不出一月，定然还清，从粉板上拭去了孔乙己的名字。'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'孔乙己\\n\\n孔乙己是站着喝酒而穿长衫的唯一的人。他身材很高大;青白脸色，绉纹间时常夹些伤痕;一部乱蓬蓬的花白的胡子。穿的虽然是长衫，可是又脏又破，似乎十多年没有补，也没有洗。\\n他对人说话，总是满口之乎者也，教人半懂不懂的。因为他姓孔，别人便从描红纸上的“上大人孔乙己”这半懂不懂的话里，替他取下一个绰号，叫作孔乙己。\\n\\n孔乙己一到店，所有喝酒的人便都看着他笑，有的叫道，“孔乙己，你脸上又添上新伤疤了!”他不回答，对柜里说，“温两碗酒，要一碟茴香豆。”便排出九文大钱。他们又故意的高声嚷道，“你一定又偷了人家的东西了!”孔乙己睁大眼睛说，“你怎么这样凭空污人清白……”“什么清白?我前天亲眼见你偷了何家的书，吊着打。”\\n孔乙己便涨红了脸，额上的青筋条条绽出，争辩道，“窃书不能算偷……窃书!……读书人的事，能算偷么?”接连便是难懂的话，什么“君子固穷”，什么“者乎”之类，引得众人都哄笑起来：店内外充满了快活的空气。听人家背地里谈论，孔乙己原来也读过书，但终于没有进学，又不会营生;于是愈过愈穷，弄到将要讨饭了。\\n幸而写得一笔好字，便替人家钞钞书，换一碗饭吃。可惜他又有一样坏脾气，便是好喝懒做。坐不到几天，便连人和书籍纸张笔砚，一齐失踪。如是几次，叫他钞书的人也没有了。孔乙己没有法，便免不了偶然做些偷窃的事。但他在我们店里，品行却比别人都好，就是从不拖欠;虽然间或没有现钱，暂时记在粉板上，但不出一月，定然还清，从粉板上拭去了孔乙己的名字。'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# chunksize == 500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size = 500,\n",
    "    chunk_overlap  = 0,\n",
    "    length_function = len,\n",
    ")\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "chunks = text_splitter.split_text(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chunk #0 , size:157\n",
      "-----------------------------------------------------------------\n",
      "chunk #1 , size:462\n",
      "-----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for i , _ in enumerate(chunks):\n",
    "    print(f\"chunk #{i} , size:{len(chunks[i])}\")\n",
    "    print(\"-----------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# chunksize == 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size = 200,\n",
    "    chunk_overlap  = 0,\n",
    "    length_function = len,\n",
    ")\n",
    "\n",
    "chunks = text_splitter.split_text(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chunk #0 , size:157\n",
      "-----------------------------------------------------------------\n",
      "chunk #1 , size:149\n",
      "-----------------------------------------------------------------\n",
      "chunk #2 , size:146\n",
      "-----------------------------------------------------------------\n",
      "chunk #3 , size:165\n",
      "-----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for i , _ in enumerate(chunks):\n",
    "    print(f\"chunk #{i} , size:{len(chunks[i])}\")\n",
    "    print(\"-----------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# chunksize ==100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size = 100,\n",
    "    chunk_overlap  = 0,\n",
    "    length_function = len,\n",
    ")\n",
    "\n",
    "chunks = text_splitter.split_text(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chunk #0 , size:3\n",
      "-----------------------------------------------------------------\n",
      "chunk #1 , size:81\n",
      "-----------------------------------------------------------------\n",
      "chunk #2 , size:70\n",
      "-----------------------------------------------------------------\n",
      "chunk #3 , size:99\n",
      "-----------------------------------------------------------------\n",
      "chunk #4 , size:50\n",
      "-----------------------------------------------------------------\n",
      "chunk #5 , size:99\n",
      "-----------------------------------------------------------------\n",
      "chunk #6 , size:47\n",
      "-----------------------------------------------------------------\n",
      "chunk #7 , size:99\n",
      "-----------------------------------------------------------------\n",
      "chunk #8 , size:66\n",
      "-----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for i , _ in enumerate(chunks):\n",
    "    print(f\"chunk #{i} , size:{len(chunks[i])}\")\n",
    "    print(\"-----------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size = 100,\n",
    "    chunk_overlap  = 0,\n",
    "    length_function = len,\n",
    "    separators = [\"\\n\\n\", \"\\n\", \"。\", \"\"] ,\n",
    "    keep_separator=  True,\n",
    "    \n",
    "    is_separator_regex = True\n",
    "\n",
    ")\n",
    "\n",
    "chunks = text_splitter.split_text(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chunk #0 , size:3\n",
      "-----------------------------------------------------------------\n",
      "chunk #1 , size:81\n",
      "-----------------------------------------------------------------\n",
      "chunk #2 , size:70\n",
      "-----------------------------------------------------------------\n",
      "chunk #3 , size:74\n",
      "-----------------------------------------------------------------\n",
      "chunk #4 , size:75\n",
      "-----------------------------------------------------------------\n",
      "chunk #5 , size:98\n",
      "-----------------------------------------------------------------\n",
      "chunk #6 , size:48\n",
      "-----------------------------------------------------------------\n",
      "chunk #7 , size:99\n",
      "-----------------------------------------------------------------\n",
      "chunk #8 , size:66\n",
      "-----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for i , _ in enumerate(chunks):\n",
    "    print(f\"chunk #{i} , size:{len(chunks[i])}\")\n",
    "    print(\"-----------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chunk #0 , context:孔乙己\n",
      "-----------------------------------------------------------------\n",
      "chunk #1 , context:孔乙己是站着喝酒而穿长衫的唯一的人。他身材很高大;青白脸色，绉纹间时常夹些伤痕;一部乱蓬蓬的花白的胡子。穿的虽然是长衫，可是又脏又破，似乎十多年没有补，也没有洗。\n",
      "-----------------------------------------------------------------\n",
      "chunk #2 , context:他对人说话，总是满口之乎者也，教人半懂不懂的。因为他姓孔，别人便从描红纸上的“上大人孔乙己”这半懂不懂的话里，替他取下一个绰号，叫作孔乙己。\n",
      "-----------------------------------------------------------------\n",
      "chunk #3 , context:孔乙己一到店，所有喝酒的人便都看着他笑，有的叫道，“孔乙己，你脸上又添上新伤疤了!”他不回答，对柜里说，“温两碗酒，要一碟茴香豆。”便排出九文大钱。\n",
      "-----------------------------------------------------------------\n",
      "chunk #4 , context:他们又故意的高声嚷道，“你一定又偷了人家的东西了!”孔乙己睁大眼睛说，“你怎么这样凭空污人清白……”“什么清白?我前天亲眼见你偷了何家的书，吊着打。”\n",
      "-----------------------------------------------------------------\n",
      "chunk #5 , context:孔乙己便涨红了脸，额上的青筋条条绽出，争辩道，“窃书不能算偷……窃书!……读书人的事，能算偷么?”接连便是难懂的话，什么“君子固穷”，什么“者乎”之类，引得众人都哄笑起来：店内外充满了快活的空气。\n",
      "-----------------------------------------------------------------\n",
      "chunk #6 , context:听人家背地里谈论，孔乙己原来也读过书，但终于没有进学，又不会营生;于是愈过愈穷，弄到将要讨饭了。\n",
      "-----------------------------------------------------------------\n",
      "chunk #7 , context:幸而写得一笔好字，便替人家钞钞书，换一碗饭吃。可惜他又有一样坏脾气，便是好喝懒做。坐不到几天，便连人和书籍纸张笔砚，一齐失踪。如是几次，叫他钞书的人也没有了。孔乙己没有法，便免不了偶然做些偷窃的事。\n",
      "-----------------------------------------------------------------\n",
      "chunk #8 , context:但他在我们店里，品行却比别人都好，就是从不拖欠;虽然间或没有现钱，暂时记在粉板上，但不出一月，定然还清，从粉板上拭去了孔乙己的名字。\n",
      "-----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for i , _ in enumerate(chunks):\n",
    "    print(f\"chunk #{i} , context:{chunks[i]}\")\n",
    "    print(\"-----------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
